Image-based deck verification

ABSTRACT

A method for preparing a deck for a process is disclosed. The deck can be prepared with any necessary components, and then an imaging device can capture an image of the deck. This image can be compared with a reference image and any differences identified. The differences can be indicated in the image and shown to an operator, such that the operator can correct any errors associated with the differences.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/966,560, filed Jul. 31, 2020, which application is a U.S. NationalStage Filing under 35 U.S.C. 371 from International Application No.PCT/US2019/015980, filed on Jan. 31, 2019, and published as WO2019/152604 on Aug. 8, 2019, which application claims priority to U.S.Provisional Patent Application No. 62/625,191, filed Feb. 1, 2018, thecontents of which are incorporated by reference in their entirety forall purposes.

BACKGROUND

Sample processing systems can be used to analyze biological samples.Once a sample processing system is programmed and the necessarymaterials are laid out in a specific way, the system can automaticallyexecute analyses or other processing for the samples. For example,biological samples in test tubes can be assembled and staged in aspecified position on a deck. Also, pipette tips can be assembled andstaged in another position on the deck and test containers such asmicrowell plates may be positioned at yet another position on the deck.Then, when an analysis process has started, a robotic arm can retrieveone or more pipette tips and use the tips to transfer a portion of thesamples from some of the test tubes as needed and transport them to thetest containers for further processing.

While these automated systems can efficiently analyze biological sampleswhen properly set up, they rely on human operators to properly preparethe system before the analysis process is started. If a human operatorplaces an array of test tubes in the wrong area, or forgets to preparethe test tubes, the system may not be able to complete the analysisprocess and/or it may cause the system to malfunction. Such human errorsare common, especially for systems that are frequently reconfigured forrunning different experiments, as well as for processes that involvemultiple types of components that need to be placed in particularlocations. Accordingly, human preparation errors frequently causeautomated biological sample analysis processes to fail.

Some techniques have been used to address this human error problem. Forexample, grid-like illustrations and virtual representations are used toshow the operator where different components should be placed in astaging area. However, even with such tools, operators continue to makemistakes, as they have trouble relating illustrations to the real-worldenvironment. For example, the operator may place components in a similarpattern, but with each component accidentally shifted to the side by onelocation. Additionally, scanners have been introduced that move fromstation to station, checking for the type of component placed in eachstation to ensure that each required component has been prepared andplaced in its proper location. However, the scanning process can taketoo much time to efficiently check the entire staging area. Accordingly,there is still a need for an improved method of preparing components anda deck staging area.

Further, alternative versions of some components may be acceptable forsome techniques. A fully automated scanning or recognition system forcomponents may identify the acceptable alternative components asdifferent from the requirements of a protocol. This may cause the systemto needlessly abort a protocol that could have been run successfully.There is thus a place for human judgment in an improved method forpreparing or verifying components within a deck staging area.

Embodiments of the invention address these and other challenges,individually and collectively.

BRIEF SUMMARY

Some embodiments of the invention incorporate an imaging device into aprocessing system. The imaging device captures an image of a deck orother staging area. When an experiment or other process is initiallyconfigured, the imaging device can capture an image of the correctlyprepared deck. At a later time, when the operator is preparing the deckfor another execution of the process, the imaging device can captureanother image. The second image can be compared with the first image toidentify any differences between the images. The differences can beindicative of human errors in preparing the deck, such as a package ofcomponents being placed in the wrong area. The real-world image of thedeck can be modified to highlight the differences to the operator, andthe operator can proceed to inspect the highlighted locations on thedeck and make any necessary corrections. The real-world imagehighlighting the differences is easier for the operator to understand,and as a result, the operator can have a high likelihood of successfullycorrecting the errors.

One embodiment of the invention is directed to a method comprisingcausing an imaging device to capture a second image of a deck of astructure. The deck comprises a plurality of discrete deck locationswith a plurality of different components respectively in the discretedeck locations. The method also includes comparing the second image to afirst image stored in a memory in a computer apparatus, determining ifthere are any differences between the second image and the first image,and outputting an indication of any differences between the second imageand the first image that may cause disruption or failure of a processrun on an apparatus.

Another embodiment of the invention is directed to a system configuredto perform the above-described method.

These and other embodiments of the invention are described in furtherdetail below, with reference to the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a processing system according to anembodiment of the invention.

FIG. 2 shows a diagram of a deck according to embodiment of theinvention.

FIG. 3 shows a flowchart illustrating a deck verification processaccording to embodiments of the invention.

FIG. 4 shows an example reference image of a deck according to anembodiment of the invention.

FIG. 5 shows an example of a comparison between a reference image and anew image according to an embodiment of the invention.

FIGS. 6A-6B show an example of different deck verification views thatcan be available to the operator, according to an embodiment of theinvention

DETAILED DESCRIPTION

Embodiments of the invention can be used to prepare a deck or stagingarea for an analysis or other process. For example, after a humanoperator places different components in different places on the deck, animaging device can capture an image of the prepared deck. This image canbe compared with another image that shows a correct setup of the deck.The differences between the current image and the previous image can behighlighted and shown to the operator. The operator can then inspect, inthe real-world, areas that correspond to the highlighted areas of theimage and make any necessary corrections (e.g., swapping an incorrectcomponent for a correct component).

The image can be captured and analyzed quickly, facilitating a fastcorrection process. Additionally, the operator can understand areal-world image better than a deck illustration, and thus can correctany errors quickly and effectively.

Prior to discussing specific embodiments of the invention, some termsmay be described in detail.

A “component” may include a part, a piece, or an element. Components caninclude tools and constituent elements used to complete a procedure,such as biological experiment or a manufacturing process. Examples ofcomponents include supplies used in biological experiments, such assample tubes, pipette tips, biological samples, reagents, chemicals,microwell plates, and any other suitable material or labware. Componentscan sometimes be grouped together in a tray or other suitable package.In some embodiments, a certain type of component can be typically foundin a uniquely configured package, or in a package with a specific label.

FIG. 1 shows a high-level block diagram of a processing system 100according to an embodiment of the invention. The processing system 100comprises a control computer 108 operatively coupled to a structure 140,a transport device 141, a processing apparatus 101, and an imagingdevice 107. Input/output interfaces may be present in each of thesedevices to allow for data transmission between the illustrated devicesand any external devices. An exemplary processing system is the Biomeki7 Automated Workstation marketed by the Beckman Coulter, Inc. of Brea,Calif.

Embodiments of the invention can include imaging a deck in order todetermine whether or not the components have been correctly arranged onthe deck. For explanatory purposes, the processing system 100 willmainly be described as a sample processing system for processing andanalyzing biological samples. However, embodiments can apply to anyother suitable type of process that involves a deck with pre-loadedcomponents.

The structure 140 may include support legs, a power source, a deck 105,and any other suitable feature. The deck 105 can include a physicalsurface (e.g., a planar physical surface) upon which components can beplaced and accessed for experiments, analyses, and processes. In someinstances, the deck 105 may be a floor or a tabletop surface. The deck105 can be subdivided into a plurality of discrete deck locations forplacing different components. The locations can be directly adjacent, ormay be spaced apart. Each deck location can include dividers, inserts,and/or any other support structure for separating the different decklocations and containing components. For exemplary purposes, FIG. 1shows a first location 105A, a second location 105B, and a thirdlocation 105C on the deck, though additional locations can be included.

The transport device 141 (which can represent multiple transportdevices) can prepare and/or transport components between the deck 105and the processing apparatus 101, as well as between different locationson the deck 105. Examples of transport devices may include conveyors,sample tracks, pick and place grippers, laboratory transport elementsthat can move independently (e.g., pucks), robotic arms, and other tubeor component conveying mechanisms. In some embodiments, the transportdevice 141 includes a pipetting head configured to transfer liquids.Such a pipetting head may transfer liquids within removable pipette tipsand may include grippers suitable for grasping or releasing otherlabware, such as microwell plates.

The processing apparatus 101 can include any number of machines orinstruments for executing any suitable process. For example, theprocessing apparatus 101 can include an analyzer, which may include anysuitable instrument that is capable of analyzing a sample such as abiological sample. Examples of analyzers include spectrophotometers,luminometers, mass spectrometers, immunoanalyzers, hematology analyzers,microbiology analyzers, and/or molecular biology analyzers. In someembodiments, the processing apparatus 101 may include a sample stagingapparatus. A sample staging apparatus can include a sample presentmentunit for receiving sample tubes with biological samples, a samplestorage unit for temporarily storing sample tubes or sample retentionvessels, a means (or device) for aliquotting a sample (such as analiquottor), a means for holding at least one reagent pack comprisingthe reagents needed for an analyzer, and any other suitable features.

The imaging device 107 can be any suitable device for capturing an imageof the deck 105 and any components on the deck 105 (or the entirestructure 140). For example, an imaging device 107 can be any suitabletype of camera, such as a photo camera, a video camera, a threedimensional image camera, an infrared camera, etc. Some embodiments canalso include three dimensional laser scanners, infrared lightdepth-sensing technology, or other tools for creating a threedimensional surface map of objects and/or a room.

The control computer 108 can control the processes run on the processingsystem 100, initially configure the processes, and check whether acomponent setup has been correctly prepared for a process. The controlcomputer 108 can control and/or transmit messages to the processingapparatus 101, the transport device 141, and/or the imaging device 107.The control computer 108 may comprise a data processor 108A, anon-transitory computer readable medium 108B and a data storage 108Ccoupled to the data processor 108A, one or more input devices 108D, andone or more output devices 108E.

Although the control computer 108 is depicted as a single entity in FIG.1, it is understood that the control computer 108 may be present in adistributed system or in a cloud-based environment. Additionally,embodiments allow some or all of the control computer 108, theprocessing apparatus 101, the transport device 141, and/or the imagingdevice 107 to be combined as constituent parts in a single device.

The output device 108E may comprise any suitable devices that may outputdata. Examples of an output device 108E may include display screens,speakers, and data transmission devices.

The input device 108D may include any suitable device capable ofinputting data into the control computer 108. Examples of input devicesinclude buttons (e.g., a keyboard and mouse), touchscreens, touch pads,microphones, etc.

The data processor 108A may include any suitable data computation deviceor combination of such devices. An exemplary data processor may compriseone or more microprocessors working together to accomplish a desiredfunction. The data processor 108A may include a CPU that comprises atleast one high-speed data processor adequate to execute programcomponents for executing user and/or system-generated requests. The CPUmay be a microprocessor such as AMD's Athlon, Duron and/or Opteron; IBMand/or Motorola's PowerPC; IBM's and Sony's Cell processor; Intel'sCeleron, Itanium, Pentium, Xeon, and/or XScale; and/or the likeprocessor(s).

The computer readable medium 108B and the data storage 108C may be anysuitable device or devices that can store electronic data. Examples ofmemories may comprise one or more memory chips, disk drives, etc. Suchmemories may operate using any suitable electrical, optical, and/ormagnetic mode of operation.

The computer readable medium 108B may comprise code, executable by thedata processor 108A to perform any suitable method. For example, thecomputer readable medium 108B may comprise code, executable by theprocessor 108A, to cause the processing system 100 to perform a methodincluding causing an imaging device to capture a second image of a deckof a structure, the deck comprising a plurality of discrete decklocations, a plurality of different components respectively in thediscrete deck locations, comparing, the second image to a first imagestored in a memory in a computer apparatus; determining if there are anydifferences between the second image and the first image; and outputtingan indication of any differences between the second image and the firstimage that may cause disruption or failure of a process run on anapparatus.

The computer readable medium 108B may comprise code, executable by thedata processor 108A, to receive and store process steps for one or moreprotocols (e.g., a protocol for analyzing a biological sample), as wellas to control the structure 140, the transport device 141, and/or theprocessing apparatus 101 to execute the process steps for the one ormore protocols. The computer readable medium 108B can also include code,executable by the data processor 108A, for receiving results from theprocessing apparatus 101 (e.g., results from analyzing a biologicalsample) and for forwarding the results or using the results foradditional analysis (e.g., diagnosing a patient). Additionally, thecomputer readable medium 108B may comprise code, executable by the dataprocessor 108A, for comparing two images of a deck, identifyingdifferences between the two images, and displaying the images withindicated differences to a user.

The data storage component 108C may be internal or external to thecontrol computer 108. The data storage component 108C may include one ormore memories including one or more memory chips, disk drives, etc. Thedata storage component 108C may also include a conventional, faulttolerant, relational, scalable, secure database such as thosecommercially available from Oracle™ or Sybase™. In some embodiments, thedata storage 108C may store protocols 108F and images 108G.

The protocols 108F in the data storage component 108C may includeinformation about one or more protocols. A protocol can includeinformation about one or more processing steps to complete, componentsused during the process, a component location layout, and/or any othersuitable information for completing a process. For example, a protocolcan include one or more ordered steps for analyzing a biological sample.A protocol can also include steps for preparing a list of componentsbefore starting the process. The components can be mapped to specificlocations on the deck 105 where the transport device 141 can obtain thecomponents in order to transport them to the processing apparatus 101.This mapping can be encoded as instructions for operating the transportdevice 141, and the mapping can also be represented by a virtual imageshown to a user such that the user can place the components on the deck105. Embodiments allow the processing system 100 to be used for multipleprocesses (e.g., multiple different biological analyses). Accordingly,information about multiple protocols 108F can be stored and retrievedwhen needed. Components on the deck 105 can be rearranged, changed,and/or replenished as necessary when changing from a first process to asecond process, or when re-starting a first process.

An image can include a depiction of one or more objects. As examples,images can include digital pictures or photographs, videos,three-dimensional pictures and videos, color photos, black and whitephotos, high dynamic range images (e.g., combining multiple images takenof the same subject with different exposures), etc. The images 108G inthe data storage 108C may include a real-world visual representation ofthe deck 105. In each image, the deck 105 can be shown in a ready statefor beginning a certain process, with all necessary components placed intheir proper locations. Each of the images 108G can be associated with aspecific protocol from the stored protocols 108F. In some embodiments,there may be a single image for certain protocol. In other embodiments,there can be multiple images (e.g., from different angles, withdifferent lighting levels, or containing acceptable labwaresubstitutions in some locations) for a certain protocol. The images 108Gcan be stored as various types or formats of image files including JPEG,TIFF, GIF, BMP, PNG, and/or RAW image files, as well as AVI, WMV, MOV,MP4, and/or FLV video files.

As mentioned above, the deck 105 can be subdivided into a plurality ofdiscrete deck locations for staging different components. The discretelocations may be of any suitable size. An example of a deck 105 with aplurality of locations is shown in FIG. 2. The deck 105 in FIG. 2 showsseparate areas numbered P1 through P30, as well as TL1 through TL5, TR1,and a wash station, each of which can operate as separate locations forseparate types of components or packages of components. Some of theseparate area numbers in FIG. 2 are obscured by overlying components:these locations may be identified by numbers sequential to thoseassociated with visibly numbered separate areas. Embodiments allow thedeck 105 to have additional locations or fewer locations as desired.While these locations can be numbered or named, they may or may not bephysically labeled or marked on the deck 105 in the real world.

Embodiments allow some or all of the locations to be occupied by apre-defined type of component according to a certain protocol. As anexample, FIG. 2 shows locations P2, P3, P4, P5, P16, P23, P24, TL4, andTL5 each being loaded with packages of components as specified by afirst protocol. Some of the locations can include the same type ofcomponent. For example, locations P2, P3, and P4 all include thecomponent type labeled BC230, which can represent a certain type of testtube, microwell plate, pipette tips, or any other suitable labwarecomponent.

In some embodiments, one or more locations may not be physically a partof the structure 140 or deck 105, but may instead be on the floor oranother surface adjacent to the structure 140 and/or the deck 105. Theselocations may be included because they can still be accessed by thetransport device 141. For example, locations TL1 through TL5, TR1,and/or the wash station may be physically separate from the structure140 and/or the deck 105.

FIG. 3 shows a high level flowchart illustrating a deck verificationprocess according to embodiments of the invention.

At step 302, during configuration of a new protocol (e.g., referred toas the first protocol), an operator can physically configure the layoutof a first set of components on the deck. For example, the operator candetermine where to place a first type of components (e.g., labware suchas pipette tips) in one or more first locations, where to place a secondtype of components (e.g., labware such as sample tubes) in one or moresecond locations, and so forth. For example, the processing apparatusand deck can be used for executing multiple processes, and theprocessing apparatus, transport device, and/or control computer can beprogrammed to allow a user to configure types of components that are tobe located in a plurality of discrete deck locations so that differentprocesses can be run on the apparatus.

At step 304, the operator can program the first protocol based on theestablished location configuration of the components. For example, thecontrol computer (and/or processing apparatus) can receive, from theoperator, information about the specific locations in which differenttype of components will be placed for use during a first process. Thiscan be done using a menu that allows a user to drag and droprepresentations of components into specific locations. The operator canalso program the steps of the first process (e.g., the steps needed toperform a certain type of biological sample analysis), and the manner inwhich the components are to be used during the first process.Embodiments allow this step of protocol programming to take place eitherbefore or after the first image is captured at step 306.

At step 306, the control computer can cause the imaging device (e.g., acamera) to capture a first image of the deck of the structure. The imagecan be taken from a static camera position and used for later reference.The image may be captured at a moment in time when the deck has beenfully loaded with the components, and when the first process performedin step 308 has not yet begun (e.g., such that the components have notyet been used, moved, or otherwise disturbed). Thus, the image cancapture a plurality of discrete deck locations as well as a firstplurality of different components respectively in the discrete decklocations.

At step 308, the control computer can execute the first process asdefined by the first protocol. For example, the transport device cantransport the components from the deck to one or more instruments of theprocessing apparatus according to the first process, and the processingapparatus can manipulate the components to perform the first process, aswell as communicate any results back to the control computer. Forexample, the processing apparatus can be an analyzer, and the firstprocess can include analyses of biological samples. Step 308 can takeplace after step 306, such that the image reflects the deckconfiguration as it appears before the process begins.

At step 310, the control computer can store (e.g., in the data storageor other memory) the first image of the deck. The first image can bestored as associated with the first protocol and used as a referenceimage for future executions of the first process. In some embodiments,the operator and/or control computer can determine to store the firstimage after step 308 if the first process is successfully completed. Ifthe first process is not successfully completed, the first image usedwith the unsuccessful first process can be discarded, the componentlayout on the deck can be reconfigured and/or the first processadjusted, and another first image can be captured. Once the firstprocess has been successfully completed, the first protocol can be savedin the data storage memory along with a captured first imagecorresponding to the successful first protocol.

At step 312, at a later time, the control computer can receive aselection of the first protocol. For example, the output device canprovide a selection window for the operator to select among variousprotocols, and the operator may select (e.g., via the input device) thefirst protocol in order to run the first process. This step can takeplace after a different process has been run on the processing system.

At step 314, the control computer can, via the output device, display afirst protocol deck configuration as specified by the selected firstprotocol. The display can include a virtual representation of the deck,the different deck locations, and the components to be placed at eachlocation. For example, the display can be similar to FIG. 2. In someembodiments, the control computer can also display the stored imagealong with any other suitable type of information for instructing theoperator how to prepare the system.

At step 316, the operator or a machine under the operator's control canphysically place a second set of components on the deck. The operatorcan use the protocol instructions and displayed component diagram as aguide for placing specific types of components in specific locations. Ifthe operator correctly places the components, each of the locations willhave the same type of component as previously staged during the initialconfiguration of the protocol (e.g., in step 302).

At step 318, the control computer can cause the imaging device tocapture a second image of the deck of the structure. In someembodiments, the second image of the deck is captured after the operatoris finished placing the second set of components, and before the processis executed. Thus, the second image can capture the plurality ofdiscrete deck locations as well as a second plurality of differentcomponents respectively in the discrete deck locations. The second imagecan be taken with the same camera, from the same position and angle,and/or otherwise taken in the same manner as the first image. As aresult, the second image and first image can be compared to identifydifferences. In some cases, multiple images from different angles orperspectives can be taken.

At step 320, the control computer can retrieve the stored first imagefrom the data storage (or other memory) based on the selection of thefirst protocol.

At step 322, the control computer can compare the first image to thesecond image in order to identify any relevant differences. For example,the control computer can identify, for each location on the deck,whether components shown in the second image are different thancomponents shown in the first image. In some embodiments, the controlcomputer can analyze the pixels or objects in each image to identifydifference in colors, shapes, component labels, or any other suitableindicators. These differences may be indicative of incorrect componentsin the second image. In some embodiments, the control computer candetermine, based on the image characteristics (e.g., colors, shapes,etc.), the actual types of components in each image and determinewhether the components are the same or different. Additional detailsregarding the image comparison are discussed below with respect to FIG.5.

At step 324, the control computer can, based on the image comparison,determine if there are any differences (e.g., one or more differences)between the first image and the second image. Additionally, the controlcomputer can determine whether any locations are not visible in theimages (e.g., due to the camera position and angle).

In some embodiments, whether or not a difference exists between imagescan be based upon a threshold of similarity. For example, if portions oftwo images are 98% similar, then this may be sufficient for the controlcomputer to conclude that there is no meaningful difference.

At step 326, the control computer can display or otherwise output (e.g.,via the output device) an indication of any determined differences tothe operator. Such differences could cause disruption or failure of theprocess if the process apparatus is allowed to run without correction.The output can include the first image and/or the second image, as wellas difference indicators on the images. For example, components andlocations in question can be highlighted, circled, or otherwiseidentified for the operator. Indicators can show the operator, forexample, a deck location that has different components in the secondimage and first image. The output might also include the virtualrepresentation from step 314.

If no differences were found, the method can continue to step 330 wherethe process is performed. For example, the output can indicate thatthere are no differences between the second image and the first imagethat may cause disruption or failure of the process run on theapparatus, and then the operator can initiate execution of the firstprocess.

At step 328, the operator can physically correct one or more componentconfiguration errors based on the displayed differences. Displaying thefirst and or second image to the user with highlighted differences canfacilitate a better operator response, as real-world images can beeasier for a person to understand than a virtual representation. Thedifferences in the images may not all require correction. For example,the correct type of component may have been used, but it may be flaggedbecause it had a different color (which may not affect the process).Accordingly, the operator may make one or more corrections (e.g.,swapping certain components in certain locations), but may not changethe components at every location with a difference indicated in theimages.

At step 330, the processing system can perform the first process usingthe second components placed on the deck. The process may be completedsuccessfully as the image comparison may have enabled the operator toidentify and correct any preparation errors.

Embodiments allow the deck setup to be verified before each subsequentrun of the first process and/or alternative processes. For example,steps 312-330 can be repeated each time the first process is beingexecuted.

As described with respect to step 306, the imaging device can captureand store the first image of the deck to be used as a reference forcomparing with a later image. An example of such a reference image 410is shown in FIG. 4, according to an embodiment of the invention. Thereference image 410 can be taken from any suitable perspective, such asa perspective that optimizes the view of the components, or from aposition where the imaging device is least likely to be disturbed. Thereference image 410 can include some or all deck locations (e.g.,depending on the camera angle, position, and field of view). It ispossible that some locations can be shielded from view by components orother obstructions. In some embodiments, multiple reference images canbe taken from multiple perspectives in order to show all the locationsand components. Additionally, multiple images can be combined togetherto create one larger image with more inclusive perspective.

The components shown in the reference image 410 can be distinguishedbased on the size, shape, color (e.g., a pipette tip rack color), welldensity, number of stacked plates (e.g., one location can include threestacked trays of pipette tips), and/or any other suitable physicalfeature of the component or the component's container. Additionally, thecomponent containers may include labels, barcodes, or other identifiablefeatures.

Because the reference image 410 can be a real-world image, the referenceimage 410 can include objects other than the components and deck. Forexample, the background of the image can include other lab equipment,walkways, shadows, and personnel, etc. that may be present in the image.In some embodiments, such additional information can be shaded orotherwise ignored by the control computer during the comparison process.This can prevent the control computer from finding and displayingirrelevant differences in the background when comparing the referenceimage 410 with a subsequent image.

FIG. 5 shows an example of a comparison between a first image 510 (e.g.,a reference image) and a second image 520 (e.g., a new image) accordingto an embodiment of the invention. The display shown to a user (e.g., asdescribed above with respect to step 326) can include one or both ofthese images.

As shown in FIG. 5, one or more locations can be indicated in the firstimage 510 and/or second image 520. The locations can be indicatedbecause they include different content in the first image 510 and secondimage 520. For example, a first difference 531 between the first image510 and second image 520 can be indicated because that deck locationdoes not have the same appearance in the two images. Similarly, a seconddifference 532, a third difference 533, and any other suitable number ofdifferences can be indicated.

In FIG. 5, the differences are indicated by superimposing a rectangularframe over the deck location. Embodiments allow alternative indicationsto be used, such as highlights, arrows, coloring, increased brightness,listing of the locations, and/or any other suitable indication. In someembodiments, the second image 520 can be made partially transparent andthen superimposed over the first image 510, such that non-matching partsof the images do not blend together and thus become visibly evident to aviewer. Further, when the first image 510 and the second image 520 areshown in the same space in this manner, a slider tool may be used toadjust how intensely either image is shown. For example, sliding to theleft can cause the first image 510 to be shown more vividly and thesecond image 520 to be more transparent, while sliding to the right cancause the second image 520 to be shown more vividly and the first image510 to be more transparent.

In some embodiments, some or all of the indicated differences can beindicative or problems with the deck setup (e.g., incorrect componentsplaced in those locations, not enough of a component a location, nocomponent in a location where there should be one, incorrect type ofcomponent at the location, etc.), and the problems can cause disruptionor failure of the process if the deck setup is not corrected.Embodiments also allow the control computer to identify imagedifferences without determining whether or not the differences wouldcause a problem. For example, the differences can represent a possibleproblem which the operator is advised to inspect (e.g., a componentpackage has a different color, which may or may not be indicative of thewrong type of component). Additionally, concerns other than differencescan be identified, such as deck locations that are not visible in theimage.

The control computer can also concurrently output a virtualrepresentation of the protocol's deck configuration, such as the displayshown in FIG. 2. This can provide the operator with an additionalperspective, and can assist the operator with identifying the type ofcomponent assigned to an indicated location. The control computer canadditionally provide a list of details related to the possible errors,the list describing the location of the possible error, as well asinformation about what led to the diagnosis of the possible error (e.g.,an unexpected color or shape)

In some embodiments, the control computer may only compare portions ofthe images that include the deck and/or components, and the controlcomputer may disregard other portions of the image (e.g., thebackground, walkway perimeters, etc.). Additionally, irrelevant portionsfirst image 510 and/or second image 520 can be erased, blacked out,shaded, masked, or otherwise obscured such that the operator's attentionis not diverted from the relevant portions (e.g., the deck andcomponents). This is shown in the figures, as the non-essentialperimeter areas of the images in FIG. 5 are darkened as compared to theoriginal reference image shown in FIG. 4.

When determining the differences between the first image 510 and thesecond image 520, the control computer can use any suitable imagecomparison technique, as well as any suitable image comparison software.In one example, the control computer can compare the color valueassigned to each pixel in both images, and identify values that differ.A difference in the images may be triggered by a single pixel differing,or by a group (e.g., 10, 20, or 100) of adjacent pixels all differing.

In some embodiments, instead of comparing small groups of pixels, thecontrol computer can identify each deck location and/or componentpackage in the images (e.g., based on a component package perimeteroutline), and can determine whether the location or component package isshown differently between the two images. Identifiable differences caninclude package color, package shape, component label or barcode, welldensity, number of stacked plates, well shapes, whether labware islidded, or any other visually identifiable difference. Some differences,such as color, can be indicative of an incorrect component, or may bethe result of the same component having a different package color.Either way, the difference can be pointed out to the operator such thatthe operator can determine whether correction is needed.

In some embodiments, the control computer can map a certain range ofpixels to a deck location. For example, the control computer can map oneof the discrete deck locations to a pixel range in the first image orthe second image. Thus, the control computer can determine whichlocation (e.g., location P14) is associated with a difference identifiedin the second image 520. This allows the control computer to label thedifferences with the location number to assist the operator withidentifying the location in question.

The control computer may be able to analyze an image and determine whattype of component is shown in each location. As a result, the controlcomputer can identify if a location includes, for example, test tubes inthe first image 510, but that same location includes pipette tips in thesecond image 520.

As an example, the first difference 531 can represent a deck locationthat is not visible in either image, the second difference 532 canrepresent a component package that has a different color or differentcomponent type in the second image 520, and the third difference 533 canrepresent a deck location that is incorrectly empty in the second image520.

In some embodiments, certain differences in component appearance can bedisregarded. For example, if not desired to be identified, the controlcomputer may not report differences in liquid color (e.g., liquid in awell, tube, or reservoir), as liquid color can change from process runto process run. Similarly, differences in the amount of a liquid can beignored, differences in hand-written information can be ignored, and anyother suitable variations that may not cause any process issues can beignored.

In some embodiments, the control computer may be able to discard thefirst image 510 and use the second image 520 as the new reference, forexample if instructed by the operator. Alternatively, the operator maychoose to add the second image 520 as another reference of an acceptabledeck in addition to the first image 510 (e.g., instead of discarding thefirst image 510). Additionally, the operator may be able to instruct thecontrol computer to accept the current setup and proceed with theprocess without making any changes to the deck.

FIGS. 6A-6B show an example of different deck verification views thatcan be available to the operator, according to an embodiment of theinvention. The control computer can output multiple displays that showimage differences and/or deck setup errors to the operator in differentmanners. The operator may be able to toggle between these differentdisplays.

FIG. 6A shows a first display which can include a first real image 610and a second real image 620. Similar to FIG. 5, the first real image 610can be a real image (e.g., taken with a camera) of a deck taken duringthe initial configuration of a process. The second real image 620 can beanother real image (e.g., taken with a camera) of a deck taken whenpreparing the deck at a later time for a subsequent execution of theprocess.

FIG. 6B shows a second display which replaces the first real image 610with a first virtual image 630. The first virtual image can include anon-real graphic depiction of the deck that demonstrates which types ofcomponents are assigned to each location. For example, the first realimage 610 can be a virtual representation of an overhead view of thedeck. Embodiments allow the first virtual image 630 to includeindications (e.g., highlights) of the deck locations which might includeerrors (e.g., based on image differences).

As a result, the operator can switch between viewing a real image of acorrect deck configuration (e.g., the first real image 610) and avirtual representation of a correct deck configuration (e.g., firstvirtual image 630). While the operator can toggle between these options(e.g., using the type toggle buttons 640), the output device cancontinually display a real image of the current deck setup (e.g., thesecond real image 620). These different images can provide the operatorwith multiple perspectives and depictions of the deck, as well asmultiple comparisons between a correct deck setup and the current decksetup, such that the operator can better understand how the deck shouldlook and exactly which deck locations might need to be corrected.

Embodiments additionally allow the operator to manipulate the display inother manners. For example, the operator can switch between differentreal images captured by different imaging devices having differentperspectives of the deck. This is shown by the perspective togglebuttons 650 which can switch between images from a left camera and aright camera. As another example, the operator may be able to zoom inand out onto a certain image, as well as reset to a full camera view.

In addition to toggling a first display area between the first realimage 610 and the first virtual image 630, the output may be configuredto switch a second display area between the second real image 620 andthe first real image 610. As a result, the operator may be able toswitch, in the same space, between the current deck setup and thecorrect deck setup. When toggling in this manner, thecorrectly-configured areas can appear the same, and the incorrectlyprepared locations can change. The display areas that change in timewhen switching between the images can be apparent to the viewer.

Additionally, in some embodiments, when the user selects or moves amouse pointer over a highlighted deck location (or a listednotification), the output device can display additional informationconcerning that location, such as the expected type of component forthat location. Further, the display can include a list of notifications(e.g., image differences or errors), and if the operator selects anentry in the notification list, the images can be modified to focus onthe location associated with the notification (e.g., by zooming in onthe deck location and/or choosing an optimal camera perspective).

Embodiments of the invention provide a number of advantages. Forexample, embodiments advantageously provide a user/operator with anintuitive and understandable tool for identifying and correcting errorsrelated to the preparation of components on a deck structure. Humansoften have difficulty understanding written instructions, alphanumericpart numbers, virtual representations of physical spaces and grids, andother typical sources of information instructing how to prepare aprocessing system for executing a process. As a result, human operatorsoften make errors when loading components and otherwise setting up aprocessing system, and this causes the process to fail or be delayed.Embodiments of the invention overcome this problem by incorporating animaging system, taking images of the deck at different times, andanalyzing the images for disparities. For example, embodiments includetaking a real image of a correctly-prepared system, taking another realimage at a later time when the system is prepared for a second time, andthen finding differences between the images. This provides the humanoperator with a marked-up photograph that can be quickly and easilyreviewed and understood, and thereby enables the operator to correct anddifferences with a high rate of success. Since any errors can becorrected before the process begins, the process can be completed with ahigh rate of success and a low rate of failure and delays.

The above description is illustrative and is not restrictive. Manyvariations of the invention will become apparent to those skilled in theart upon review of the disclosure. The scope of the invention should,therefore, be determined not with reference to the above description,but instead should be determined with reference to the pending claimsalong with their full scope or equivalents.

One or more features from any embodiment may be combined with one ormore features of any other embodiment without departing from the scopeof the invention.

A recitation of “a”, “an” or “the” is intended to mean “one or more”unless specifically indicated to the contrary.

All patents, patent applications, publications, and descriptionsmentioned above are herein incorporated by reference in their entirety.

What is claimed is:
 1. A method of processing a sample using anautomated liquid handler, the method comprising: a) receiving, using aninput device of the liquid handler, a selection from a user of aprotocol for processing the sample; b) displaying, on an output deviceof the liquid handler, a graphical depiction of an expected type oflabware component to be loaded onto a deck of the liquid handler at aspecified location according to the selected protocol; c) receiving,onto the deck of the liquid handler, a loaded labware component placedat the specified location by the user; d) imaging, using an imagingdevice, the deck of the liquid handler, the image capturing thespecified location; e) identifying in the image, by a processor of theliquid handler, the type of labware component loaded onto the deck atthe specified location by the user; f) displaying an indication, by theliquid handler, to the user, if the identified type of labware componentloaded onto the deck at the specified location is different than theexpected type of labware component at the specified location accordingto the selected protocol; and g) processing, by the liquid handler, thesample according to the selected protocol.
 2. The method of claim 1,wherein the type of labware component loaded onto the deck at thespecified location is identified based on a distinguishing physicalfeature of the labware component.
 3. The method of claim 1, wherein thestep of processing the sample according to the selected protocolincludes using the loaded labware component placed at the specifiedlocation by the user if the identified type of labware component placedat the specified location is the same as the expected type of labwarecomponent at the specified location according to the selected protocol.4. The method of claim 1, further comprising receiving, using the inputdevice, input from the user, confirming that the type of labwarecomponent placed at the specified location is in fact the expected typeof labware component at the specified location according to the selectedprotocol, wherein the step of processing the sample according to theselected protocol includes using the loaded labware component placed atthe specified location by the user.
 5. The method of claim 4, whereinthe input from the user, confirming that the type of labware componentplaced at the specified location is in fact the expected type of labwarecomponent, is received after the liquid handler displays the indicationthat the identified type of labware component loaded onto the deck atthe specified location is different than the expected type of labwarecomponent
 6. The method of claim 1, further comprising receiving, ontothe deck of the liquid handler, a swapped labware component placed atthe specified location by the user to replace the loaded labwarecomponent, wherein the step of processing the sample according to theselected protocol includes using the swapped labware component at thespecified location.
 7. The method of claim 6, wherein the swappedlabware component is placed at the specified location by the user afterthe liquid handler displays the indication that the identified type oflabware component loaded onto the deck at the specified location isdifferent than the expected type of labware component.
 8. The method ofclaim 1, wherein the step of processing the sample according to theselected protocol includes using the loaded labware component placed atthe specified location by the user if the processor determines that theloaded labware component is an acceptable alternative labware componentat the specified location according to the selected protocol.
 9. Themethod of claim 1, wherein the graphical depiction of the expected typeof labware component to be loaded onto the deck includes a virtualrepresentation of the deck, the different deck locations, and thelabware components to be placed at each location according to theselected protocol.
 10. The method of claim 9, wherein the indication isdisplayed on the virtual representation of the deck.
 11. The method ofclaim 1, wherein the image captures a plurality of discrete decklocations, and a corresponding plurality of different labwarecomponents, respectively loaded onto the deck at the plurality ofdiscrete deck locations.
 12. The method of claim 1, wherein the imagecaptures the entire deck.
 13. The method of claim 1, wherein the step ofprocessing, by the liquid handler, the sample according to the selectedprotocol includes using a pipetting head of the liquid handler totransfer liquids into or out of the loaded labware component placed atthe specified location by the user.
 14. The method of claim 13, whereinthe sample comprises a biological sample.
 15. The method of claim 1,further comprising displaying, on the output device, additionalinformation concerning the expected type of labware component for thespecified location according to the selected protocol.
 16. The method ofclaim 1, wherein the type of labware component loaded onto the deck atthe specified location is identified based on any one or more of thesize, shape, color, well-density, or stacking of the loaded labwarecomponent.
 17. An automated liquid handler for processing a sample, theautomated liquid handler comprising: a) a computer apparatus configuredto control the processes performed by the automated liquid handler, thecomputer apparatus including a processor, an input device coupled to theprocessor, and an output device coupled to the processor, wherein theprocessor is configured to receive, via the input device, a selection,from a user, of a protocol for processing the sample by the automatedliquid handler, wherein the processor is further configured to display,on the output device, a virtual representation of an expected type oflabware component to be loaded onto a deck of the liquid handler at aspecified location according to the selected protocol; and b) an imagingdevice coupled to the processor, the imaging device configured tocapture an image of a labware component loaded onto the deck, by a user,at the specified location, for processing the sample using the selectedprotocol, wherein the processor is further configured to identify, inthe image, the type of labware component loaded onto the deck, by theuser, at the specified location, and wherein the processor is furtherconfigured to display, on the output device, an indication that the deckmay be incorrectly loaded if the identified type of labware componentloaded onto the deck at the specified location is different than theexpected type of labware component according to the selected protocol.18. A method of preparing a deck for a process, the method comprising:loading a protocol comprising at least one of: information about aprocessing step used to execute the process; and a component locationlayout on a deck in a state ready to execute the process according tothe protocol; displaying a stored image of at least a portion of thedeck in a state ready to execute the process according to the protocol;capturing an image of a placed component on the deck; identifying theplaced component in the image; and indicating whether the placedcomponent is at least one of a correct component and properly located onthe deck in order to execute the process according to the protocol. 19.The method of claim 18, further comprising determining whether theplaced component in the captured image is the same type of component asa component displayed in the stored image.
 20. A method for initiatingan instrument, the method comprising: receiving a protocol for executinga process; displaying a stored image of a deck with a component placedat a location, based on the protocol; capturing an image of the deck;and indicating a placed component on the deck.
 21. The method of claim20, further comprising using a processor to identify, in the capturedimage, the type of component placed on the deck.